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(54) High resolution camera 

(57) A digital camera has an image sensor and an 
optical system for focusing a scene on the image sen- 
sor. A zooming system is used for changing a zoom rate 
of the scene that is focused on the image sensor By 
operating the zooming system during the recording of a 
series of images, each of the images has a different 
zoom rate and therefore a different level of detail and 
field of view. The images are combined to form one still 
image of super high resolution. The zoom rate between 
images is detected by using horizontal and vertical pixel 
intensity value projection distributions in both fonvard 
and backward prediction methods. Horizontal and verti- 
cal candidate projection distributions that are correlated 
to a range of possible zoom rates are compared with the 
horizontal and vertical projection distributions of the 
former and cun-ent images being processed to deter- 
mine a least enror match that is selected as the zoom 
rate between the acfjacent images. Using the image to 
image zoom rate, a scaling factor is determined that is 
used for scaling each curent image before ovenmriting 
the cun'ent image over the former images previously 
written to the image buffer. In this way, the detail of each 
image in the series is written to the image buffer and a 
super high resolution, wide angle still image is formed 
that can be stored and displayed. 
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Description 

Reld of the Invention 

[0001] The invention relates to a digital camera that 
processes serial inriages to obtain a wide angle still 
infiage with high resolution. In particular, a super high 
resolution digital camera produces a wide angle still 
image with high resolution from images input by a 
zooming operation. 

Background of the Invention 

[0002] For photography and related uses, many types 
of digital cameras have t>een sold in recent years. For 
achieving the objects of portability, functionality, or pic- 
ture quality, various technical features have been 
adopted. Since the photographed image is digital, the 
image can be directly input into a computer, transmitted 
between computers and published, for example on the 
homepage of an Internet site or with a printer. There- 
fore, the demand for digital cameras has been growing 
quickly. However, the picture quality of present digital 
cameras is greatly inferior to that of conventional photo- 
graphic film (silver salt) cameras. The resolution (typi- 
cally 320 pixels 240 pixels) of the present average 
digital camera limits the ability of a user to enlarge the 
angle of view of the captured image. 
[0003] Therefore, a need exists for a still image digital 
camera which provides a high resolution image even 
when a wide angle photograph is sought. 
[0004] In Sallient Video Stills: Content and Context 
Preserved. ACM Multimedia 93 pp. 39-46 written by 
Laure Tedosic et al, a method for obtaining an image 
with high resolution by using several conventional 
images is disclosed. This camera uses an affine trans- 
formation as a model for the camera's operation, and 
calculates 6 parameters from a group of local motion 
vectors between continuous Images and synthesizes 
one wide angle image with high resolution from the plu- 
ral Images. 

[0005] On the other hand, in the field of coding 
images, "A Study about the Alignment in Sprite Gener- 
ation" (IE97-81(1997) pp. 19-24) written by Hataizumi et 
al is known which discloses a method for generating a 
wide angle still image known as "sprite". This metiiod 
uses Hehnert's transformation as a model of the canrv 
era's operation, and calculates a pan, a zoom and a 
rotation parameter of the camera, and synthesizes one 
wide angle image from the plural images using these 
parameters. Likewise with tiie aforementioned first 
method, each parameter is calculated from tiie group of 
local motion vectors between continuous images. 

Summary of the Invention 

[0006] Generating a digital still image witti high reso- 
lution in a prior art digital camera requires the calcula- 



tion of local motion vectors between (continuous or 
serial) images. In order to calculate a motion vector, a 
search process is executed by block matching of blocks 
of a size of 16 X 16 pixels. As a result, tiie block match- 

5 ing needs to be calculated for hundreds of blocks in tiie 
Image. Therefore, there is a problem with increasing tiie 
image resolution because of the corresponding require- 
ment of greatiy increasing the quantity of calculation. 
[0007] Furthermore, since the camera operation 

10 related parameters are estimated based on local motion 
vectors, for tiie case where a moving object exists in tiie 
image, tiiere is a problem ttiat the moving image causes 
an error in the resolution of the image. Therefore, it is 
difficult to generate an image with high resolution by 

15 applying conventional methods to a digital camera. 
Accordingly, in tiie prior art, a practical problem exists 
with the detection of camera operation related peirame- 
ters in real time in order to make the detection perform- 
ance quick and efficient. 

20 [0008] It is an object of tiie present invention, there- 
fore, to provide a digital camera which generates a wide 
angle still picture with high resolution similar to that 
obtained by conventional wide angle photography at a 
low cost, and in real time from a series of input images. 

25 preferably input by recording a scene with the camera. 
[0009] The camera of tiie invention receives a series 
of input images, preferably by capturing the input 
images witii an image sensing system for photograph- 
ing an arbitrary scene, such as a moving or dynamic 

30 image. For recording a series of images, the camera 
has an operation input switch for starting and ending the 
recording of a scene, during which tiie camera's zoom 
function is operated to either zoom in for enlarging the 
image (conrespondingly reducing the angle of view) of 

35 the scene being shot or to zoom out for reducing tiie 
image (conrespondingly widening tiie angle of view) of 
tiie scene. 

[0010] The camera also preferably has a display for 
displaying the high resolution Image as it is being 

40 recorded and after processing. Preferably, the display is 
made a part of tiie camera body for a user to view tiie 
image during recording. Optionally, the image taken in 
by tiie view finder can be included as part of tiie display. 
Also, the display can show a graphical user interface 

45 that shows image information related to the zooming 
operation displayed at the same time on the display witti 
the images being recorded. 

[0011] As part of tiie processing of tiie series of 
images, a zoom rate detector detects a zoom rate 

50 between images, preferably images photographed by 
the image sensing system of tiie camera or otiienvise 
input to the camera from storage, for example. The 
zoom rate detected between each of the images is 
related to a scaling factor that is used to enlarge or 

55 reduce sequential images In the series during tiie 
processing of ttie images whrch are then stored in an 
image buffer to obtain tiie super high resolution image. 
[0012] For example, an Initial image of a series of 
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images is written to the image buffer. Then each suc- 
ceeding next image is multiplied by the appropriate scal- 
ing ^ctor determined in relation to the detected zoom 
rate and ovenwritten on the image buffer to provide for 
greater Image detail of that part of the scene which is 
captured by each next image of the series. This process 
is repeated in real time during the recording operation of 
the camera, with each next image of the series (for each 
sequential image of the series or for every other image 
thereof, etc.), being scaled up or scaled down according 
to the zoom rate detected by the zoom rate detector and 
then being overwritten on the image buffer. 
[0013] During processing, as the series of images is 
being recorded and ovenvrltten on the image buffer, 
preferably the resultant image is displayed on the dis- 
play in real time. After the recording of the series of 
images is concluded, or othenwise input to the camera, 
and all of the images to be processed have been appro- 
priately scaled and ovenwritten on the Image buffer, the 
resultant image In the image buffer is the super high res- 
olution image. The super high resolution image can be 
displayed and stored as a typical digital image in the 
camera in an image memory and othenArise output for 
storage In an external memory or display. 

Brief Descri ption of the Drawinns 

[G0141 

Fig. 1 shows an example of a camera system of the 

invention. 

Fig. 2A shows an example of the arrangement of 
the camera body of the invention. 
Fig. 2B is an example of a camera display 120 
shown In Fig. 2A. according to an alternative 
embodiment of the invention. 
Fig. 2C is another alternative embodiment of the 
display 120 shown in Fig. 2A. 
Fig. 3 Is a diagram schematically illustrating the 
process of forming a still image with super high res- 
olution according to the present invention. 
Fig. 4 is a diagram Illustrating the combining of 
serial images according to the present invention. 
Fig. 5 is a diagram showing zoom rate detection 
between two sequential images. 
Fig. 6 Is a diagram of a comparison made between 
a vertical projection distribution 503 obtained from 
an input image for matching with a candidate verti- 
cal projection distribution in the determlhatidh of a 
zoom rate. 

Figs. 7A and 7B are examples of graphical inter- 
faces for users that displays a rate of zoom for a 
sequence of Input Images according to a prefen-ed 
embodiment of the Invention. 
Fig. 8 shows an example of the flowchart of a cam- 
era control program. 

Fig. 9 is a diagram of image buffers and data struc- 
tures for forming a super high resolution image 



according to the present Invention. 
Rg. 10 shows an example of zoom rale detection 
between images for generating super high resolu- 
tion Image. 

5 Rg. 11 shows an example of the flowchart of a 
process of detecting a zoom rate between sequen- 
tial images from vertical projection distributions of 
the Images. 

Rg. 12 shows an example of the flowchart for a 
10 process of detecting a zoom rate between sequen- 
tial Images from horizontal projection distributions 
of the images. 

Detailed Description of the Preferred Embodiments 

75 

[0015] Fig. 1 shows a digital camera system according 
to the invention that processes serial images preferably 
input while recording the images during a zooming 
operation to obtain a wide angle still image with super 

20 high resolution. 

[001 6] The present application Is copending with U.S. 
application serial no. 09/093,782, filed June 9, 1998 
entitled Digital Wide Camera, the disclosure of which is 
hereby incorporated by reference. 

25 [0017] The digital camera of the invention is function- 
ally divided into two parts and may be formed In a single 
camera body or housing, or may be divided into two or 
more parts. If divided, the optical, audio and image 
sensing parts of the camera and an image display, as 

30 well as the image compression and decompression 
coding circuits, can be separate from the Image 
processing part of the camera which processes the 
series of images to obtain the s\xper high resolution 
image, optionally stores the serial images as well as 

35 optionally displays the images. The images can be input 
by a recording operation of the camera for real time 
processing during the recording or be input from stor- 
age, optionally external of the camera. 
[0018] As shown In Fig. 1, a bus 108 schematicaUy 

40 depicts a connection between the components of the 
camera. Above the bus 108 are shown components that 
are generally equivalent to the components found In a 
conventional MPEG camera, such as the MPEG- 10 dig- 
ital disk camera of Hitachi, Ltd. Below the bus 108 are 

45 shown the components typically found In a digital com- 
puter, and in particular, according to one embodiment of 
the invention, a notebook computer. Note, however, that 
an image display 120 for displaying a processed 
sequence of images as a super high resolution image 

50 122 is shown below bus 108, but in a preferred embod- 
iment, as shown In Figs. 2A -2C, the Image display Is 
formed as part of the camera body so that, preferably, 
formation of the image as the camera is processing the 
serial Images can be observed by the user. 

55 [0019] In particular, the digital camera includes an 
optical system 100 including a view finder 101. which 
may be an optical view finder. Alternatively, the view 
normally provided by the view finder 101 may be pro- 
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vided as an image on the display 120 that is displayed 
along with the super high resolution Image 122 or as an 
alternative thereto. Of course, display 120 can also dis- 
play other suitable information, shown for example in 
Figs. 2B and 2C. to be explained in greater detail here- 
inafter. 

[0020] In more detail with reference to Fig. 1 . optical 
system 100 is used for focusing and framing the view of 
a scene or person for capturing the view on an image 
sensing device (CCD) 102 under control of the user who 
observes the scene or person through the optical sys- 
tem finder 101. The optical system includes a niagnrfi- 
cation adjusting mechanism for operating the zoom 
function. The operation data such as the zoom (magni- 
fication), etc. is transmitted to a CPU 1 12 that performs 
functions to be described later. 
[0021] In general, when a scene is shot using an 
MPEG camera, the images are converted to electric sig- 
nals through photo-electric conversion by image sens- 
ing device 102. An image compression encoder 104 
receives the electric signals representing an image from 
the image sensing device 102. which reduces the 
amount of data by using an image compression method 
such as that of the MPEG standard. The images can be 
stored in an external storage device 110. which can 
either be part of the camera, for example a flash mem- 
ory or a hard disk drive, or can be external to the cam- 
era, for example a memory storage device in a 
computer. 

[0022] The images are taken In at a frequency of 
about 30 times per second (30 frames per second} so 
that they are stored as consecutive images, each hav- 
ing a potentially different frame to frame zoom rate. 
According to the prefenred embodiment of the invention, 
the series of images are processed by enlarging or 
redudng the captured image in accordance with a scal- 
ing factor determined on the basis of the frame to frame 
zoom rate detected between adjacent frames or images 
of the series to be processed. After scaling an image, 
the image is stored or preferably ovenwritten onto an 
image buffer (super high resolution image buffer). The 
storing starts with writing an initial image to the image 
buffer based on a determination of whether the series of 
images represents a zoom in or zoom out operation of 
the camera during recording. Then, each succeeding 
image of the series is scaled according to the zoom rate 
with respect to the previous image and then written over 
the previous image in the high resolution image buffer. 
[0023] Although each image or frame can be used at 
this rate while still accomplishing real time processing, 
every other or every third (and so on) frame at this rate 
can alternatively be selected as the series of images to 
be processed. Also, a higher rate of images can be cap- 
tured by the recording system in the event that different 
standards, such as MPEG 2. for example, are adopted 
for use. The rate at which Images are selected to make 
up a series of images is determined, according to the 
present invention, so that the detail of the resultant 



image Is satisfactorily maintained. 
[0024] When a still Image is to be captured, i.e. an 
image without increased resolution achieved by 
processing a series of images using the zoom function, 

5 only one image at the moment when a shutter is 
pressed has to be stored. Thus, the camera can func- 
tion as a standard digital camera. 
[0025] Sound can be captured along with the recorded 
scene with a microphone 126 and an A/D converter 

10 128. The sound is preferably stored in the external stor- 
age device 1 10 and correlated to the input images. 
[0026] Preferably, the external storage device 1 10 is a 
large-capacity storage device such as a hard disk or the 
like that provides a non-volatile storage area for digital 

75 data. The hard disk may be of the type which can be 
detached from the main txxJy like a hard disk caid of the 
PCMCIA type or it may be of the type in which only the 
storage medium is detached, like a magneto-optical 
disk. 

20 [0027] The CPU 1 1 2 executes the control operation 
for starting/stopping the recording operation and further 
executes the software program for realizing the func- 
tions which will be described in the present invention. A 
program, such as a camera control program 114-1. 

25 resides in a memory 1 14, in which are also stored val- 
ues of variables and image data (data 1 14-2. showwi in 
greater detail in Fig. 9) that are necessary for the execu- 
tion of the program. 

[0028] An image memory or video memory 1 1 6 stores 

30 a super high resolution image 122 to be displayed on a 
display 120, as digital data. A D/A converter 118 of a 
type which is generally called a RAMDAC successively 
reads the data written on the video memory 116. 
adjusts to the speed of the scanning lines, and draws an 

35 image on the display 120. When the data in the video 
memory 1 16 is updated, therefore, the updated content 
is readily reflected on the display 120. The image being 
shot and input to the image compression encoder 104 
can also be directly written In the video memory 1 16 by 

40 bypassing the Image compression function. 

[0029] The Image decompression decoder 106 
decompresses the compressed dynamic image data 
stored in the external storage device 1 10 and outputs it 
to the video memory 1 16 as an image. An analog output 

45 1 24 outputs the deconpressed image after converting it 
into analog video signals of the type widely used for TV 
receivers of, e.g.. the NTSC system. These outputs can 
be recorded by connecting the camera to a video appa- 
ratus such as that of a VHS type that is now widely 

50 used. The audio data that is recorded can be output 
through an audio D/A converter 132 and a loudspeaker 
130 in synchronism with the decompression of the 
image. An input l/F 123 can be used for the control of 
switches such as a record button and a replay button, or 

55 it can accommodate a data input device such as a 
transparent touch panel attached to the surface of the 
display 120. The input data are transmitted to the CPU 
112 and are suitably processed. 
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[0030] In general, for the digital camera of this embod- 
iment, a sequence of images that are shot while operat- 
ing the zoom in/ zoom out function are processed to 
obtain a super high resolution image. Rrst, the images 
that are shot are fed to the memory 1 1 4-2 according to 5 
a camera control program stored in the memory 1 14-1 
duhrKf the shooting, and then are successively proc- 
essed to display a high resolution image 122, which dur- 
ing processing as shown In Fig. 1, includes a wide angle 
Image 122-1 of the just processed image and a current 10 
(zoomed-in) image 122-2 on the display 120. After the 
shooting is finished, the wide image is stored as an 
image data structure 110-1 in the external storage 
device 110. 

[0031] With reference to Fig. 1. each image data is 
structure includes an image header 110-1-1 that has 
data necessary for the decompression, such as the 
image compression system used, and the^vertical and 
lateral sizes of the image. A flag 110-1-2 which repre- 
sents whether the image is valid or invalid, is used at the 20 
time of simply editing the images. The image data size 
110-1-3, which represents the number of bytes of the 
succeeding image data 110-1-4, is also stored. There- 
fore it is possible to calculate where the compressed 
image data are recorded in accordance with the kind of 25 
the image or the size of the image. 
[0032] The display 120 is a device for displaying an 
image, and may be. for example, a small CRT or a 
plasma display or may be a display device of the liquid 
crystal type. In addition to displaying the image stored in 30 
the video memory 1 16. additional images or alternative 
images can be displayed on display 120, such as the 
view from the optical system finder 101. 
[0033] Fig. 2A illustrates the appearance a digital wide 
camera body according to a preferred embodiment of 35 
the invention. In this embodiment, a camera body 200 
has the aforementioned lens unit (optical system) 100, 
finder unit for the optical system 101 , flat panel display 
1 20, and buttons 20 1 to 205. The image shot by the lens 
unit 100 can be confirmed through the finder unit 101 40 
and can be displayed on the display 120. 
[0034] Of the buttons 201-205, an operation switch 
201 is for zoom control, which is a rocker button switch 
in which the zoom operation is continuously performed 
when the switch is depressed toward one side or the 45 
other for increasing or decreasing the magnification of 
the viewed scene. The recording is accomplished upon 
depressing the record button 202; i.e.. the recording 
starts when it is depressed once and ends when it is 
depressed again during recording (or only while it is so 
being hekJ down, etc.). In adcfition to recording the 
Images shot during the recording period, when the user 
operates the camera to shoot a scene while operating 
the zoom function, a high resolution wide Image is 
formed in real time by processing tiie image sequence. 55 
and Is displayed on the display 120. This makes it pos- 
sible to quickly confirm the processed image and to 
obsen^e potential failure in the processing of a wUe 



image. 

[0035] When the playback of the stored images is 
requested by operating a button, a list and the individual 
images are displayed on the screen 120. Further, for the 
camera of this embodiment, general-purpose buttons 
203 and 204 are also provided for enabling various 
operations of tfie camera. Still further, a scroll button 
205 is provided for selecting an Image to be displayed 
and for saolling through the stored images to be dis- 
played on tfie display 120. 

[0036] The recorded images can be output through an 
analog output 124 after being converted Into analog 
video signals that are widely used for TV receivers. Fur- 
thermore, the recorded data can be transferred to a 
general computer through a computer interface 206. 
Thus, the digtal wide camera of the present invention 
can be effective as a tool for collecting a variety of digital 
based information. 

[0037] Although the present embodiment contem- 
plates that the images of a zoom sequence are input 
through the lens by the operation of the camera, tiiey 
may be input to auxiliary storage 110 from another 
source. Then, such a sequence of images can be read 
from the auxiliary storage separately and processed to 
generate the super high resolution image. 
[0038] Figs. 2B and 2C show alternative en*odiments 
of the arrangement of display 120. contemplated by ttie 
present invention. In fig. 2B. the display 1 20 indtdes an 
input image display portion 122-3 that shows tiie image 
directiy from the view finder 100 and further shows the 
super high resolution image as it is being processed in 
display area 122-4. Also, a graphical interface display 
area 122-5 is provided for displaying graphical informa- 
tion relating to the degree of zoom for the current image 
relative to the other images of tiie sequence being proc- 
essed, preferably in real time as explained witti refer- 
ence to Figs. 7A and 7B. 

[0039] The an^angement of the display 1 20 in the other 
alternative embodiment shown in Fig. 2C is similar to 
the Fig. 28 embodiment in tiiat the display includes 
input image display portion 122-3 tiiat shows tiie Image 
directly from the view finder 100. On the other hand, the 
graphical user interface display area 122-6 shown in 
tiiis figure is different. In particular, the change in zoom 
magnification is shown by changing the size of a border 
122-7 displayed as part of tiie high resolution image, 
wherein the border 122-7 is around tiie periphery of tiie 
current image in the sequence of images being proc- 
essed, and contacts or expands as shown by the 
anrows. depending on the direction of the zoom in the 
sequence. In a preferred embodiment, the border is dis- 
played by showing a line of a particular color as the bor- 
der 122-7, such as a highlighting color like yellow. 
[0040] Fig. 3 is a diagram illusb^ating the steps for 
forming a super high resolution still image which will 
now be described, also with reference to Fig. 4. Once 
tiie record button 202 is depressed, a sequence of pic- 
tures or images (frames) are created beginning with a 
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first image 121 -1 of the sequence at time Is and contin- 
uing until the record button 202 is depressed again at 
time Te. The picture sequence 121 that is shown in Fig, 
3 is obtained when the pictures are taken while operat- 
ing the camera's zoom in function, in this example. The 
zoom function causes the scene in the images of the 
sequence to be reduced or enlarged in small incremen- 
tal changes, depending on the actual frame to frame 
zoom rate in operation. 

[0041 ] That is. a wide view image of mountains, trees, 
houses and a helicopter is shot at the start of recording 
the sequence at Ts and an image of just the helicopter, 
one tree and part of the mountains is shot at the end of 
recording the sequence at Te. During this zoom-in 
sequence, the part of the wide view captured by the 
image increases in detail as a result of the zoom in func- 
tion, but the angle of view decreases. 
[0042] Preferably, to capture the sequence of images, 
the camera is mounted on a tripod or other fixed support 
so that during the zoom function there is substantially lit- 
tle shifting of the center point of each respective image 
in the sequence. For example, the center point is fixed 
at the point of the intersection of Wy2, H/2 for each 
image, wherein the images have a height H and a wWth 

as shown in Fig. 5. 
[0043] However, the rate of zoom in or zoom out is 
rarely constant so the frame to frame zoom rate fluctu- 
ates between images in the sequence. Even under such 
a situation, by scaling the images according to the zoom 
rate between each Image processed in a series of 
images, a super high resolution image 122 can be 
obtained. As shown in Fig. 4. the zoom rates between 
inoages are determined (400) by doing a check based 
on an image feature quantity between adjacent (in time) 
images 121-1 and 121-2. The latest image, scaled 
according to the determined zoom rate, is over written 
on the previously processed series of images stored in 
a super high resolution image txjffer fbmiing the super 
high resolution Image 122-1 (step 410) in real time until 
the end of processing of the series of images after 
which the final super high resolution image 122 is 
obtained. 

[0044] Although a process of generating a super high 
resolution image when the series of images are cap- 
tured in a zoom-in operation has been explained herein, 
the process is the same for the zoom out operation. In 
the case of zoom out, by scaling the latest image, only 
the area that extends into an area beyond that of the 
former image is over written on the super high resolu- 
tion image buffer to obtain the super high resolution 
image 122-1 (410) for efficiency. 
[0045] Here, scaling of an Image can be simply per- 
formed by a method of expanding or contracting an 
image according to a scale, or by applying a method of 
linear interpolation by refening to 4 pixels in a circumfer- 
ence. While the method of linear interpolation has a 
problem due to the increase by 4 times of the calcula- 
tion quantity, it can make the jag of an image un-con- 



spicuous even when a super high resolution image is 
expanded and displayed. When the generated super 
high resolution image is displayed in real time, it is desir- 
able that the size of the first image at the time of the 

5 start of recording is known in advance according to the 
lens magnification. . . 

[0046] In a preferred embodiment of the invention, the 
direction of the zoom function, i.e. zoom in or zoom out, 
operated during the recording of the images is deter- 

10 mined first. The first image of a series of images to be 
processed is considered to be one half of the maximum 
magnification or zoom rate. Then. tNs first image is 
compared with the second image of the series to deter- 
mine whether the operator is zooming in or zooming out 

15 during recording. If the operator is zooming in during 
recording, then the first image is scaled and stored at 
the maximum size of the super high resolution image 
fcHJffer 114-2-2 (Fig. 9) using the appropriate scaling fac- 
tor as associated with the camera's maximum zoom 

20 rate. Then, the next image is scaled (enlarged or 
reduced in accordance with the detection of the frame to 
frame zoom rate) and is ovenwritlen on the image Ixjffer 
1 14-2-2 with a center point matching between the two 
images. The center point is assumed to be fixed for 

25 each frame or image at W/2, H/2 according to this 
embodiment of the invention. The second and each 
subsequent image is then multiplied by a scaling foctor. 
which is determined on the basis of the image to image 
(frame to frame) zoom rate for each subsequent image 

30 with respect to the preceding image and followed by 
ovenvriting the image on the image buffer for the high 
resolution image, as explained in greater detail herein. 
[0047] For the case of zoom in. the center part of the 
image stored in the image buffer 1 14-2-2 is ovenwritten 

35 each time by the next succeeding image, which lias 
been multiplied by a decreasing scaling factor or zoom 
rate. For example, if the maximum zoom rate is 10 for 
the recording system and the operator is zooming In on 
the scene during recording, the first image is written to 

40 the video buffer using a scaling factor of 10, which is 
incrementally reduced for each succeeding image 
according to the actual frame to frame (image to image) 
zoom rate, from 10 for the first image to a last image in 
the series having a lowest possible scaling factor d 1 .0. 

45 The scaling factors are not preset since each is deter- 
mined in real time on the basis of the actual frame to 
frame zoom rate, which is determined within a tolerance 
of about 0.001 to 0.005 and preferably within a toler- 
ance of about 0.001 . 

so [0048] As another consideration, since the center por- 
tion of the super high resolution image buffer is overwrit- 
ten for each image, it is quicker to ovenwrite only a 
peripheral edge portion of each succeeding image, 
rather than the entire image. To perform this overwriting . 

55 function according to the prefenred embodiment of the 
invention, a variable sized center portion of each suc- 
ceeding image is masked to alleviate the need for over- 
writing the center most portion of the image to the 
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case where the zoom function has not been used 
between images 12M and 121-2, in which case the 
candidate vertical distribution of projection 501, which 
represents a zoom rate of 1 .0 will be determined as the 
candidate vertical distribution of projection that matches 5 
503. This would indicate a still picture (still meaning no 
zoom b^een images 12M and 121-2) image 121-2. 
[0059] Although only the comparison of vertical pro- 
jection distributions is shown in Fig. 6. horizontal projec- 
tion distributions are also compared and the nearest 10 
match selected in the same way. According to the pre- 
ferred embodiment of the invention, both the vertical 
and horizontal projection distributions are compared to 
ensure accurate determination of the zoom rate. 
[0050] Further, although Fig. 6 only shows the con- is 
cept of fonward prediction, preferably the zoom rale is 
also determined using backward prediction. In back- 
ward prediction, the candidate vertical distribution pro- 
jections are generated based on the image 121-2 at 
time Ti+1 for comparison with vertical distribution of pro- 20 
jection 501. That is, the candidates are generated with 
iterations of a zoom rate expressed by (W/(W+n)). By 
using both fonward and backward prediction techniques 
with both horizontal and vertical distributions of projec- 
tion, the zoom rate can be d^ermined with high accu- 25 
racy. 

[OCSI] At the time of detecting the zoom rate, since 
scaled candidate projection distrifc>utions of an image 
are checked, rather than using a scaled image directly, 
it is possible to achieve a dramatic reduction in the 30 
quantity of calculations performed In determining the 
zoom rates. TTierefore, If the CPU has about 100 mips 
performance, this processing can be done thirty times 
or more in 1 second, and this makes it possible to real- 
ize the super high resolution image processing of the 35 
invention in real time. 

[00S2] As mentioned above, the reason why the zoom 
rate is preferably detected on the basis of both the ver- 
tical projection distribution and a horizontal prqection 
distribution is that an error in determining the zoom rate 40 
is less likely. An en^or nrray be potentially caused by the 
movement of an c^ject which is resolved in the vertical 
direction only or the horizontal direction only. For exam- 
ple, when a scene is being recorded as a series of 
images shows a door that is open which is then shut, 45 
the zoom direction determined by using the vertical pro- 
jection distribution results in a determination of zoom in 
when the door is "open". On the other hand, the zoom 
direction is determined to be zoom out when the door Is 
"shut". In contrast, accading to the horizontal projection so 
distribution, the zoom is determined to be still, which is 
connect- Therefore, an accurate zoom prediction is made 
using both the horizontal and vertical projection distribu- 
tions and selecting the one among all of the candidates 
that has the best match as a result of the comparison, ss 
[0CS3] Incidentally, the distribution of projection of an 
image is found by normalizing. Normalizing in the verti- 
cal direction means to sum the intensity values of the 



pixels in each column of height H and cfivide that sum by 
the numt>er of pixels for tiie column; and normalizing in 
the horizontal direction means to sum the intensity val- 
ues of the pixels in each row W and divide the sum by 
the number of pixels in the row. This type of normalizing 
allows the invention to be applicable to different image- 
sensors, since the array of pixels to be processed can 
be made variable, and therefore can vary between the 
typical number for a digital camera which is 320 x 240 
up to a typical format of 640 x 480 without requiring fun- 
damental differences in the processing techniques. 
[0064] Since the intensity value of a pixel is a combi- 
nation of the degrees of Intensity for red, green and 
blue, it is at times convenient to use an approximation of 
the intensity value, by using only the intensity value of 
green, for example. The intensity value of green is 
approximately equal to the intensity of the pixel and can 
be used as the intensity value. However, it is preferred 
that the intensity value be calculated from tfie RGB val- 
ues when practical. 

[0065] Frg. 8 shows an example of camera control pro- 
gram 114-1 of this invention. This program flowchart 
shows mainly the process of generating a super high 
resolution image during the recording of a sequence of 
images. Of course, previously recorded and stored 
images can be processed in the same manner Addi- 
tionally, the program can include processes for editing 
the accumulated image, referencing the image and 
communicating ttie image, etc. Camera control program 
1 14-1 is executed by refemng to conti-ol data 1 14-2 that 
is shown in Fig. 9. 

[0066] Step 800 is directed to initialization processing, 
wherein ttie variable STATUS is reset to 0. While ttie 
power of tfie camera is on (step 802), ttie following 
steps are followed in processing tiie images captured by 
tiie recording system for generating a super high reso- 
lution image. 

[0067] First, in step 804. it is examined whether record 
txitton 202 has been pressed. If so, the ftow moves to 
step 806, wherein the status is determined to be either 
1 or 0 when the record button was pressed. When the 
record button has been pressed and the STATUS is 0, 
tiie start of recording processing 808 is executed. Here, 
the preceding super high resolution Image displayed on 
display 120 of ttie camera is deleted for preparation of 
ttie display of a new super high resolution image. That 
is. SUPER_RESOLUTrONjMAGE_BUF (super high 
resolution image buffer) 114-2-2 (Fig. 9) Is initialized. 
[0068] Next, ttie variable STATUS is set to 1 at step 
810 to effect the recording. When it Is determined in 
step 806 that the record button has been pressed with 
ttie STATUS being set to 1, on the other hand, a deter- 
mination that the recording has ended is made and the 
processing 812 Is executed to create header informa- 
tion for each image (frame), to evaluate ttie size of ttie 
image data and to store ttie images in storage. Here, 
ttie image header 1 10-1-1 and image data size 1 10-1-3 
of ttie images are written in a predetermined place in 
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center portion of the iniage buffer until necessary. This 
causes the processing speed to increase significantly, 
enabling the processing to be performed in real time 
during the recording of the scene. 
[0049] In performing the processing of the super high 
resolution image of the invention, it is necessary to 
know the appropriate scaling factor, which Is related to 
the frame to frame or image to image zoom rate. An 
example of determining the appropriate scaling factor 
which is used for the scaling of an image before it is writ- 
ten to the image buffer will be explained in detail with 
reference to Figs. 5 and 6. 

[0050] Fig. 5 shows a principle of detecting the zoom 
rate. First, a horizontal projection distribution 500 and a 
vertical projection distribution 501 of image 121-1 at 
time Ti are created. Similarly, horizontal projection dis- 
tribution 502 and vertical projection distribution 503 of 
image 121-2 at time Ti+1 is created. 
[0051] By checking the difference between vertical 
projection distributions 501 and 503. the zoom rate is 
predicted, i.e. zoom in or zoom out. This is performed by 
checking both a backward prediction (from Ti+1 to Ti) 
and a fonward prediction (from Ti to Ti+1). The result of 
the prediction that has the fewest errors, i.e. the candi- 
date projection distribution that produces the closest 
match, becomes the resultant candidate projection dis- 
tribution from which the frame to frame zoom rate is 
detected, as explained in greater detail hereinafter with 
respect to Fig. 6. 

[0052] In general, in performing the forward prediction 
zoom rate detection, several temporary zoom rates are 
predicted from the vertical projection distrbution 501 of 
image 121 -1 at time Ti. and a tenrtporary vertical projec- 
tion distribution for each zoom rate is generated. Then, 
the temporary vertical projection distrtoutions that are 
formed are checked for a match with vertical projection 
distribution 503 of image 121-2 at time Tt+1. 
[0053] For backward prediction of the zoom rate, sev- 
eral temporary zoom rates are predicted from the verti- 
cal projection distribution 503 of images 121-2 at time 
Ti+I , and a temporary vertical projection distribution for 
each zoom rate is generated. Then, the temporary ver- 
tical projection distributions that are generated are 
checked for a match with vertical prqection distrBxjtion 
501 of image 121-1 at time Ti. 
[0054] Since the fluctuation range of the zoom rate 
between adjacent images is limited, the temporary 
zoom rates are predicted to be within a set range. One 
of the temporary zoom rates is selected based on the 
nearest match of the tennporary vertical projection distri- 
butions that are formed and tfie vertical projection distri- 
bution 503 (fonvard prediction). The nearest match is 
determined by mathematically considering which of the 
temporary vertical projection distributions when conv 
pared with the vertical projection distribution 503 has 
the least error due to a mismatch resulting from the 
comparison. As a check en^or between those two pro- 
jectk)n distributions, I.e. the difference error as a result 



of the checking or comparing, the sum of the absolute- 
value of a difference between each elements is used or 
alternatively the average square error can be used. 
Accordingly, a sharp reduction in the calculation quan- 

5 tity can be achieved. 

[0055] Rg. 6 shows an example of using fbnA^rd pre- 
diction in determining the frame to frame zoom rate 
using the vertical distribution of projection 503 taken 
from image 121-2. At first, several candidate vertical 

10 distibulion projections 501-1 to 501-4, are generated 
for matching with vertical distribution projection 503. 
The candidate vertical distribution projection which 
matches vertical distributbn projection 503 is used for 
determining the zoom rate, preferably within a tolerance 

15 of ±0.001 to ±0.003. If there is no best match once all 
the candidate projection distributions have been consid- 
ered for both the fonward and backward prediction mod- 
els, a zoom rate of 1 .0 is assumed. 
[0056] As shown in Fig. 6, when the zoom rate is 

20 greater than 1 .0, the width (W) of the pixels for an image 
121-2 increases with respect to image 121-1. So, the 
candidate vertical distribution projections 501-2 and 
501-1 are generated by assuming that the pixel width 
(W) increases. Similarly, tine candidate vertical distribu- 

25 tion projections 501-3 and 501-4 are generated for the 
case of the zoom rate decreasing to less than 1.0 in 
which case the pixel widtfi (W) decreases. A candidate 
vertical distribution of projection 501 is provided for the 
case of a constant pixel width (W). 

30 [0057] For accuracy in determining tiie zoom rate, tiie 
candidate vertical distribution projections that are to be 
matched to the vertical distribution projection 503 are 
generated by increasing or decreasing tiie pixel widtti 
(W) by units of 0.1 to 2.0 pixels, depending on tiie pixel 

35 widtii of the inrage. In the preferred embodiment, as 
shown, one pixel is used for expansion or contraction for 
each candidate for a practical maximum of 1 6 to 32 can- 
didate vertical distribution projections. In tiie present 
embodiment, there is considered a maximum zoom rate 

40 of ((W±10)/W). However, only tiie candidates of 
(W+1)/W and (W+2)/W. 501-2 and 501-1, respectively, 
are shown for increasing zoom rate and only (W-1)/W 
and (W-2)/W. 501-3 and 501-4. respectively, are shown 
for tiie decreasing zoom rate, for purposes of Illustrating 

45 tiie concept of the prefen'ed embodiment ol the inven- 
tion. 

[0058] Once a candidate vertical distribution of projec- 
tion is determined to match (when a best mathematical 
comparison results in the least error for one of tiie can- 

50 didates) the vertical distribution of projection 503. then 
the corresponding zoom rate ((W ± n)/W) for ttie candi- 
date Is selected as the zoom rate where n = number of 
pixels that width (W) has increased (a positive number) 
or decreased (a negative number) due to tiie enlarge- 

55 ment or reduction between images resulting from tiie 
zoom function. The preferred embodiment of determin- 
ing the image to image zoom rate is described in detail 
with reference to Figs 10-12. Of course, ttiere is flie 



7 



15 



EP0940 978A2 



16 



the image data structure 110-1. Next, the STATUS is 
reset to 0 at step 814 since the recording has ended. 
[0089] Although all of the images captured during 
recorded can be stored and processed, the images can 
also be processed in real time as they are being 5 
recorded, wherein the images are stored In image buffer 
114-2-1 , one at a time for processing and then scaled 
and overwritten into image buffer 114-2-2 as each new 
image is written into the image buffer 114-2-1. 
[0070] While the power is on, the status is checked in io 
step 820 to determine whether the status is 1 or not. 
When the status is 1 because of recording, the following 
processing is executed for the images input from the 
image pick up device. At step 822. the image is input 
through the image sensing device 102 and the Image i5 
compression encoder 104, and Is stored in the 
IMAGE_BUF 114-2-1 of the memory 114, whereby the 
frame number is incremented by 1 . Next, at step 824, 
the projection distribution in the vertical and horizontal 
directions are calculated from the input image 1 1 4-2-1 . 20 
The above-mentioned distributions of projection are 
stored in X_PROJ_CURRENT 114-2-4 and in 
Y_PROJ_CURRENT 114-2-6 of Fig. 9, respectively. 
The distributions of projection are stored until the next 
frame or image is input. In such a case, the distributions 25 
are moved to X^PROJ.LAST 114-2-3 and to 
Y_PROJ_LAST 114-2-5. respectively. When yet a fur- 
ther frame is input, they are discarded. 
[0071 ] At step 826. the distribution of projection of the 
preceding frame is collated with the distribution of pro- 30 
jection of the cun^ent frame to calculate ttie zoom rate. 
This processing will be described later in detail with ref- 
erence to Fig. 10. 

[0072] At step 828, a cun^ent input image is scaled 
according to a detected zoom rate between a last one In 35 
the series of images being processed and the current 
image, and the scaled cun-ent image is ovenMritlen on 
image buffer SUPER_RESOLUTIONJMAGE_BUF 
114-2-2. 

[0073] In performing step 828. it is necessary to know 40 
the appropriate scaling tector z. which is related to the 
frame to frame or image to image zoom rate, as follows. 

Zn=Zn-lO/2bx) 

45 

where n is tiie number of iniages in the series of images 
being processed and Zt>x is flie frame to frame zoom 
rate between adjacent or time sequential Images in the 
series of images being processed. Incidentally, the 
zoom rate is defined here by a ratio that expresses how so 
much the size of an image changes between two adja- 
cent images, as determined by tiie number of pixels 
which consecutively processed images expand or con- 
tract witti respect to one another. 
[0074] Since the part of tiie image buffer correspond- 55 
ing to the center portion of the series of images being 
processed is ovenwritten each time a next succeeding 
image in the series is written to the image buffer 1 14-2- 



2. It is quicker to overwrite only a peripheral edge por-" 
tion of each succeeding image, rather than tiie entire 
image. To perform this overwriting function according to 
tiie preferred embodiment of the invention, a variable 
sized center portion of each succeeding image is 
masked to alleviate the need for ovenwriting the center 
portion of the image to tiie center portion of the Image 
buffer unless necessary to finish the image processing 
as when writing the first or last image of the series. This 
causes the processing speed to increase signif icantiy. 
[0075] At step 830, the image in 

SUPER_RESOLUTIONJMAGE_BUF 114-2-2 is trans- 
mitted to video memory 116, and a super high resolu- 
tion image is displayed on display 120. 
[0076] At the time of transfer, the content in the 
SUPER_RESOLUTIONJMAGE_BUF 114-2-2 may be 
displayed after being thinned out (selective pixel dele- 
tion) according to the resolution of the display 120. A 
display having a high resolution is generally expensive, 
tiierefore the display generally has a lower resolution 
capability than ttie super high resolution Image and the 
Image must be processed to display it on a display witti 
lower resolution. However, by preserving the contents of 
the SUPER_RESOLUTIONJMAGE_BUF 114-2-2 as it 
is. It is possible to cfisplay and print the image witii high 
resolution tiirough another display device or printer. 
[0077] Next, with reference to Fig. 10. an example of 
the detection of tiie zoom rate used in forming tiie super 
high resolution Image of the preferred embodiment of 
tfie present invention will be described. 
[0078] In step 1000. a zoom rate is detected from a 
vertical projection disti-ibutlon and tiie determined value 
is set as variable ZOOM_X. The details of tiiis process- 
ing are explained witii reference to Fig. 1 1 . Next, at step 
1 100 a zoom rate is determined from the horizontal pro- 
jection distribution and the determined zoom rate is set 
as variable ZOOM_Y. The details of this processing are 
explained witii reference to Rg. 12. 
[0079] At first. It Is unknown what the classification of 
tiie zoom rate is among tiie 3 possible cases or classes, 
i.e. 1.0 (still picture), greater ttian 1.0 (incre^e in zoom 
rate or zoom in), or less than 1 .0 (decrease in zoom rate 
or zoom out). Accordingly, in step 1200, a determination 
Is made as to whether the zoom function is zoom in 
(ZOOM^X > 1.001 and ZC)OM_Y > 1.001). zoom out 
(ZOOM^X < 0.999 and ZOOM^Y < 0.999) or still 
(ELSE). Accordingly, the direction of zoom is classified 
into one of these three classes. The zoom direction is 
determined to be zoom in when both ZOOM_X and 
ZOOM_Y exceed 1 .001 . 

[0080] The reason why 1 .001 is used instead of 1 .0 is 
tfiat an additional tolerance of. for example, 0.001 is 
given for avoiding an erroneous determination of tiie still 
dass. The tolerance is given according to the size of tiie 
image, or ttie speed of camera operation, etc. This sup- 
presses mistaken determinations ttiat might ottierwise 
arise due to image fluctuation caused by movement of 
an object in the scene being recorded, etc. 
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[0081] The zoom direction is determined to be zoom 

out if both 200M_X and 200M_Y are less than 0.999, 
which is a value also predetermined in consideration of 
a tolerance feictor. In case the result is not one of the 
above mentioned conditions, the zoom direction is 5 
determined to be neither zoom in or zoom out and 
therefore the still class is determined. 
[0082] After a determination of zoom out or zoom in 
has been made, tiie zoom rate ZOOM is assigned the 
value of either ZOOM_X or ZOOM_Y in an appropriate 10 
one of steps 1202, 1203. 1205 and 1206. That is. if 
zoom in is determined in step 1200. then in step 1201 
the value of MINVAL_X is compared witii MINVAL_Y As 
a result of the comparison, the value of ZOOM_X is 
selected as ZOOM if MINVAL_X is less than MINVAL^Y 15 
(step 1 202) and the value of ZOOM_Y is selected as the 
zoom rate if MINVAL^X is greater than MINVAL^Y (step 
1203). In this way, the smaller one of MINVAL_X and 
MINVAL_Y is selected, which indicates the least error 
resulting from tiie comparison, and is set as the value 20 
for the variable ZOOM, which becomes the zoom rate 
between images. 

[0083] A similar series of determinations are per- 
formed after it is determined that the zoom direction is 
zoom out in steps 1204, 1205 and 1206. In the case of 25 
a still zoom rate determination. 1 .0 is set as tiie variable 
ZOOM at step 1207. 

[0084] Rg. 11 shows the details of a process of 
detecting ZOOM_X and MINVAL_X from the vertical 
projection distribution using both fonA^ard and backward 30 
prediction; and Fig. 12 shows the details of a process of 
detecting ZOOM^Y and MINVAL_Y from the horizontal 
projection disti-ibution also using both fonn^ard and back- 
ward prediction used in determining the zoom rate. A 
detailed description of Fig. 11 follows with the under- 35 
standing that the desaiption of Fig. 12 is the same, 
except for the difference witii respect to the use of verti- 
cal projection distributions in Fig. 1 1 and horizontal pro- 
jection distributions in Fig. 12. In each of tiie figures, "F" 
refers to fon/vard prediction and "B" refers to backward 40 
prediction. 

[0085] In particular. Fig. 1 1 shows tfie details of deter- 
mining ZOOM_X in step 1000, shown in Fig. 10. At step 
1 001 , variable MINVAL_FJ( that stores a check en-or at 
the time of doing a forward prediction is set to a suffi- 45 
cientiy large value, tiieoreticaliy infinitely large, but prac- 
tically a large value that exceeds the maximum possible 
value of the variable. 

[0086] At step 1 002, tiie range that is supposed is set 
as a variable T sirK:e the fluctuation range of the zoom so 
rate fc>etween continuous images is limited. In this exam- 
ple, the size of fluctuation in the image size (sight) is 
based on a range from -10 pixels to +10 pixels with 
respect to the overall pixel width dimension of an image. 
[0087] At step 1003. by using the projection distribu- 55 
tion of a fomier frame stored in X_PROJ_LAST 114-2- 
3. a projection distribution when tiie size of a sight fluc- 
tuates by 0) pixels is generated and is stored in 



X_PROJ_WORK (temporary vertical projection distribu- - 
tion arrangement) 114-2-7. 

[0088] Since tiie original projection disti^ibution is 
scaled in the direction of a one-dimensional projection 
axis, the temporary projection distribution can be calcu- 
lated at high speed. Next, a check is made between tiie 
projection distribution of a present frame stored in flie 
X_PROJ_CURRENT 114-2-6 and a temporary projec- 
tion distribution stored in X_PROJ_WORK 1 14-2-7, and 
a difference error at the time of checking is stored as a 
variable ERROR (step 1004). 

[0089] At step 1005, the value of ERROR and 
MINVAL_F X is compared, and when the ERROR is 

less than MINVAL_F_X. MIN_F_DX is set to variable i, 
and the value of ERROR is substituted for MINVAL_F_X 
(Step 1006). 

[0090] By repeating the above process, when variable 
MIN_F_DX becomes i, variable MINVAL_F_X is calcu- 
lated. At step 1007, by using the value of above variable 
MIN_F_DX. the zoom rate of a fonward prediction is cal- 
culated and is stored as variable ZOOM_F_X. 
[0091 ] In tiie foregoing, it is assumed that ttie widtii of 
an image is W, and accordingly the zoom rate for for- 
ward prediction is calculated by (W+MIN_F_DX) /W. 
[0092] Step 101 1 is similar to step 1 001 and the pro- 
cedure of steps 1012-1016 and 1020 are also similar to 
steps 1002-1007, except that tiiese latter steps are 
directed to backward prediction of tiie zoom rate. First, 
variable MINVAL_B_X Is set to a sufficientiy high value 
(step 1010). and at step 1012, the fluctuation range that 
is supposed is set to variatrfe i. In this part of the flow, 
tiie size of fluctuation in tiie image size (sight) is also 
based on a range from -10 pixels to +10 pixels witii 
respect to the overall pixel widtii dimension of an image. 
[0093] At step 1013. by using the projection distribu- 
tion of a present frame stored in X_PROJ_CURRENT 
114-2-4, a projection distribution when tiie size of a 
sight fluctuates by i pixels is generated and is stored in 
buffer X_PROJ_WORK (temporary vertical projection 
distribution arrangement) 114-2-7. 
[0094] Next, a check is made between a projection 
disti'iDution of a former frame stored in buffer 
X_PROJ_LAST 114-2-3 and a temporary projection dis- 
tilbulion stored in buffer X„PROJ_WORK 114-2-7, and 
a difference en-or resulting from the check at the time is 
stored in variable ERROR. 

[0095] At step 1015. the value of ERROR and 
MINVAL_B_X is conpared, and when the ERROR is 
less tiian MINVAL^B^X, MIN_B_DX is set to variable i. 
and the value of ERROR is substituted for 
MINVAL_B_X (Step 1016). By repeating tiie above 
process, when variable MIN_B_DX becomes i, variable 
MINVAL_B_X is calculated. At step 1020, by using tfie 
value of the above variat)le MIN_B_DX. ttie zoom rate of 
a backward prediction is calculated and is stored as var- 
iable ZOOM_B_X. 

[0096] The zoom rate for backward prediction is calcu- 
lated by evaluating tiie expression W/(W+MIN_B_DX). 
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[0097] In steps 1021 to 1024, MINVAL_F_X and 
MINVAL_B_X are checked to determine if each is 
greater than a threshold, and if so. the reliability of the 
zoom detection is judged to be a bad result, and 
ZOOM_B_X is assigned the default value of 1 .0. 5 
[0098] At step 1025. MINVAL_F_X and MINVAL_B_X 
are compared and at step 1026 or 1027, the zoom pre- 
diction value of the smaller one is stored in variable 
ZOOM^X and the corresponding one of MINVAL_F_X 
and MINVAL_F_Y is stored as the variable MINVAL^X. 10 
Variable MINVAL^X Is consequently based on which- 
ever of the fonward prediction or backward prediction 
that Is determined to be more accurate and then is com- 
pared with MINVAL^Y in steps 1201 and 1204, as 
explained with reference to Fig. 10. is 
[0099] In Fig. 12. steps 1 1 00 - 11 27 respectively cor- 
respond to steps 1000 - 1027. as explained above, so 
further explanation of the corresponding steps in Fig. 12 
is unnecessary, except with respect to the following dif- 
ferences. 20 
[0100] In Fig. 12, at step 1 102. the range that is sup- 
posed is set as a variable T representing the size of 
fluctuation in the image size (sight) based on a range 
from -10 pixels to +10 pixels with respect to the overall 
pixel height dimension of an image. Further, since the 25 
zoom rate determination in Fig. 12 is based on horizon- 
tal distribution projections, the zoom rate is calculated 
assuming an image of height H. by evaluating the 
expression (H + MIN_F_DY)/H in step 1107 (fonward 
prediction) or H/(H + MIN_B_DY) in step 1120 (back- 30 
ward prediction). 

[01 01 ] The result of the flow in Fig. 1 2 is the detection 
of ZOOM_Y and MINVAL^Y, which are used in steps 
1200-1206, shown in Fig. 10. 

[0102] An example of a graphical user interface that 35 
displays graphic information related to the camera's 
zoom operation on display 120 is shown in Figs. 7A and 
78. Vertical projection distribution 700 and horizontal 
projection distribution 701 are displayed graphically in 
time transition by setting a horizontal axis to represent 40 
photographing time from the start of recording Ts to the 
end of recording Te. and a vertical axis to represent pro- 
jection position. 

[01 03] This is obtained by translating the value of each 
vertical projection distribution 50 1 and horizontal projec- 45 
tion distribution 500 in Fig.5 into a gradation of shades 
and cHsplaying one line vertically according to each 
frame. 

[0104] Figs. 7A and 7B show an example of the tran- 
sition that occurs for the projection distributions of a so 
time series in the case of photographing using the zoom 
in function. By displaying such a transition on display 
120 (Figs. 2B and 2C, for example), a user can easily 
understand the camera's zoom operation. 
[0105] When the camera's operation is different from 55 
that expected by the user's experience, a different pat- 
tern appears. For example, in the case of a still opera- 
tion, a pattern that is parallel to the horizontal axis 



appears. Accordingly, the operator is immediately- 
apprised of the difference. Moreover, in the case of pan. 
an inclined pattern that has a constant angle in the hor- 
izontal axis appears. 

[0106] The preferred embodiment of the invention 
shows a super high resolution camera that generates a 
high resolution image at the center of screen by using 
the camera's operation of zooming in or zooming out. 
[01 07] It is possible to generate an image that has a 
high resolution not only in the center of an image but 
also In another targeted area by combining a pan or tilt 
operation with zoom. For example, as shown in Flg.3. 
when a pan operation is done, by ovenwriting an image 
obtained by the pan operation on an image at the time 
Te. it is possible to generate an image with high resolu- 
tion in a broader area. 

[0108] In addition, for the case where both the zoom 
operation and the pan operation are performed, it is 
possible to generate a high resolution image. In this 
case, a high resolution image is generated by predicting 
a shift of position in addition to predicting a zoom rate. 
[01 09] The camera of this invention is not limited to the 
camera body embodiment shown in Fig. 2A. Of course, 
it is possible to connect a video camera that enables the 
photographing of a moving image to a digital computer 
such as a personal computer, to perform a process like 
the above mentioned one for an image taken from the 
video camera, and display the high resolution image on 
a display of the digital computer. 
[0110] In this case, a digital computer can store the 
camera program 1 14-1 on recording media for control- 
ling the processing of images to obtain a super high res- 
olution imaga The program can be stored on magnetic 
disK optical disk. etc. or provided through a network 
connection to which a digital computer is connected. In 
such a case, the program would be supplied to a digital 
computer, and the digital computer would read a pro- 
gram for generating a super high resolution image from 
the recording medium to the main memory. 
[01 1 1 ] It is possible that the resolution of a super high 
resolution image display is higher than that of the reso- 
lution of the image pickup device. 
[01 1 2] This invention provides a super high resolution 
camera that is able to obtain a still image with wide 
angle and high resolution comparable to that obtained 
with a wide angle film camera with low cost, high effi- 
dency, and in real time. Furthermore, the situatiwi of 
aeating a super high resolution image is confirmed on 
the display momentarily so as to provide immediate 
feedback to the user for preventing errors in the photog- 
raphy. In addition, the image can be transmitted to a 
conputer, and it is possible to watch it as a high resolu- 
tion inf>age by a high resolution printer or a high resolu- 
tion display 

[0113] While preferred embodiments have been set 
forth with specific details, further embodiments, modifi- 
cations and variations are contemplated according to 
the broader aspects of the present invention, all as 
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determined by the spirit and scope of the following 
clainns. 

Claims 

1 . A digital camera* comprising: 
an image sensor; 

an optical system for focusing a scene on the 
image sensor; 

a zooming system for changing a zoom rate of 
the scene focused on said image sensor; 
an operation input means for starting and end- 
ing recording of a scene as a series of consec- 
utive images; 

zoom rate detection means for determining an 
image to image zoom rate between ones of 
each adjacent previous Image and current 
image of said series that is due to movement of 
the zooming system during the recording; 
scaling means for scaling each said current 
image with respect to said adjacent previous 
image in accordance with image to image 
zoom rate; 

an image buffer and a display for displaying a 
super high resolution image; 
means for oventvriting each said current image 
on said image buffer after said current image 
has been scaled with said scaling means and 
means for outputting a super high resolution 
image from said image buffer to said display. 



series of images generated during the recording. 

8. A digital camera according to daim 1 . wherein said 
zoom rate detection means includes a pixel inten- 

5 sity value projection distribution determining and 
storage means tor storing distributions of projection 
in pixel intensity in horizontal and vertical directions 
of the current and previous images, and a means 
tor detecting a fractional shift in image width and 

10 height among consecutive images of the series due 
to movement of the zooming system during the 
recording. 

9. A digital camera according to claim 8, wherein sakj 
15 pixel intensity value projection distrlkxjtion deter- 
mining and storing means determines said pixel 
intensity value projection distributions by normaliz- 
ing the results of integration of the intensity values 
of pixels constituting an image in the horizontal 

20 direction and the vertical direction, respectively, 
with respect to the number of pixels in the integra- 
tion. 

10. A digital camera according to daim 1, wherein said 
25 zoom rate detection means determines an image to 

image zoom rate between said adjacent previous 
nnage and said cun-ent image using forward predic- 
tion and backward prediction wherein a least error 
result of one of the tonnard and backward prediction 
30 of the Image to image zoom rate is seleded as said 
zoom rate. 



2. A digital camera according to claim 1 , wherein said 
means for outputting a super high resolution image 
from said image buffer operates during said record- 
ing to display a super high resolution image In 
progress on said display in real time. 

3. A digital camera according to claim 1 , furth©- com- 
prising an image storage means for storing said 
super high resolution image output from said image 
buffer. 

4. A digital camera according to daim 1, further 
including sakJ display having displays said a portion 
for displaying a super high resolution innage and a 
portion for displaying a graphical user interface. 

5. A digital camera according to claim 4, wherein said 
graphical digital interface includes means for dis- 
playing a graphical Image related to movement of 
the zooming system during the recording. 

6. A digital camera according to claim 3. wherein said 
storage means Is a detachable storage medium. 

7. A digital camera according to claim 3. further 
Induding said Image storage means storing said 



11. A digital camera according to daim 1. wherein said 
zoom rate detection means determines one of 
35 zoom in, zoom out and still as a condition of the 
movement of the zooming system during the 
recording before determining an image to Image 
zoom rate between adjacent Images. 

40 12. A digital camera, comprising: 

a camera body having a finder for viewing a 
scene being recorded, an optical system and 
an Image sensing device for shooting consecu- 

45 five images of a scene received through said 

optical system during recording, a zooming 
system for changing a zoom rate of the scene 
focused on said image sensing device, said 
consecutive images being separated by an 

so Interval of time during which movement of the 

zooming system occurs, a recording operation 
switch for starting and ending recording of a 
scene and a display for displaying a super high 
resolution image; 

55 an Image buffer for storing a single one of said 

consecutive Images at a time in and an image 
display buffer for storing said super high resolu- 
tion image; 
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a processor for determining an image to image 
zoom rate between each adjacent previous 
image and current image of said series caused 
by movement of the zooming system during the 
recording, for scaling each said current image 5 
with respect to said adjacent previous image in 
accordance with said image to image zoom 
rate, for overwriting each said cun-ent image on 
said image buffer after said current image has 
been scaled within said interval of time and out- w 
putting a super high resolution image from said 
image buffer to said display for displaying said 
super high resolution image. 



13. A digital camera according to claim 12, wherein is 
said zoom rate detection means takes the projec- 
tions of the intensity values of the pixels in the hori- 
zontal and vertical directions to perform forward 
and backward prediction between said adjacent 
previous and current images to determine said 20 
image to image zoom rate caused by movement of 
the zooming system during the recording. 

14. A digital camera according to claim 12, further 
including a memory for storing a plurality of said 25 
super high resolution images. 

15. A digital camera according to claim 12. wherein 
said camera body has a graphical user interface for 
displaying graphical images related to movement of 30 
the zoon^ng system during the recording. 

16. A digital camera according to claim 12, wherein 
said camera body has an interface for connecting 
said camera body with said processor, and wherein 35 
said processor is separate from said camera body 

17. A digital camera according to claim 12. wherein 
said view finder is an optical view finder. 

40 

18. A digital camera, comprising: 



including an image display buffer which said proc- "* 
essor writes each said consecutive image to after 
scaling each said image with, wherein said display 
receives an output from said buffer to display said 
super high resolution image in real time. 

20. A digital camera according to claim 18, further 
including an image storage means for storing at 
least one said super high resolution image. 



a viewer for viewing an image to be projected 
on an image sensing device; 
an optical system through which an image is 45 
projected on an image sensing device to take a 
plurality of images of a scene, having a zoom- 
ing system; 

an input device for starting and ending record- 
ing of the plurality of images; so 
a processor that consecutively receives said 
plurality of images and sales each image 
according to an image to image zoom rate 
between each of said innages; and 
an image display that displ^s a super high res- ss 
olution image in real time. 

19. A digital camera according to claim 18, further 
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(57) A digital camera has an image sensor and an 
optical system for focusing a scene on the image sensor. 
A zooming system is used for changing a zoom rate of 
the scene that is focused on the Image sensor. By op- 
erating the zooming system during the recording of a 
series of images, each of the Images has a different 
zoom rate and therefore a different level of detail and 
field of view. The images are combined to form one still 
Image of super high resolution. The zoom rate between 
Images Is detected by using horizontal and vertical pixel 
intensity value projection distributions In both fonward 
and backward prediction methods. Horizontal and ver- 
tical candidate projection distributions that are correlat- 
ed to a range of possible zoom rates are compared with 
the horizontal and vertical projection distributions of the 
fonner and cun^ent images being processed to deter- 
mine a least error match that is selected as the zoom 
rate between the adjacent images. Using the image to 
image zoom rate, a scaling factor is determined that is 
used for scaling each current image before overwriting 
the current image over the former images previously 
written to the image buffer In this way, the detail of each 
image in the series is written to the image buffer and a 
super high resolution, wide angle still image Is formed 
that can be stored and displayed. 
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